Skip to content

feat: Add Spark SQL as a shallow query engine - #2241

Merged
ferenc-csaky merged 1 commit into
mainfrom
feat/sparksql-dialect
Aug 5, 2026
Merged

feat: Add Spark SQL as a shallow query engine#2241
ferenc-csaky merged 1 commit into
mainfrom
feat/sparksql-dialect

Conversation

@ferenc-csaky

@ferenc-csaky ferenc-csaky commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Key Changes

  • Add the sparksql query engine with Spark SQL statement generation backed by Iceberg deployment artifacts.
  • Add Spark SQL Calcite conversion and dialect support, including backtick identifiers and Spark-native scalar, temporal, array, map, and struct type rendering.
  • Refactor JDBC DDL generation to quote table, view, index, and column identifiers through the active SQL dialect.
  • Make query-engine server configuration optional so engines without runtime server configuration, such as Spark SQL, are supported.
  • Add a Spark SQL compile use case and snapshot covering primitive, temporal, binary, collection, row, and aggregate result types.

@ferenc-csaky
ferenc-csaky force-pushed the feat/sparksql-dialect branch from 2b05e18 to 4e4dd6e Compare July 29, 2026 18:25
@codecov

codecov Bot commented Jul 29, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 23.95833% with 73 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.28%. Comparing base (9b0cce9) to head (67d4d98).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...asqrl/calcite/dialect/ExtendedSparkSqlDialect.java 20.58% 27 Missing ⚠️
...engine/database/relational/ddl/CreateIndexDDL.java 0.00% 13 Missing ⚠️
...qrl/engine/database/relational/SparkSqlEngine.java 0.00% 6 Missing ⚠️
.../database/relational/PostgresStatementFactory.java 0.00% 4 Missing ⚠️
.../database/relational/SparkSqlStatementFactory.java 0.00% 4 Missing ⚠️
...e/database/relational/ddl/DdlIdentifierQuoter.java 33.33% 3 Missing and 1 partial ⚠️
...se/relational/ddl/GenericCreateViewDdlFactory.java 0.00% 4 Missing ⚠️
...ine/database/relational/SparkSqlEngineFactory.java 0.00% 3 Missing ⚠️
...e/relational/ddl/GenericCreateTableDdlFactory.java 70.00% 3 Missing ⚠️
...atasqrl/calcite/convert/AbstractSqlConverters.java 0.00% 1 Missing ⚠️
... and 4 more
Additional details and impacted files 
@@            Coverage Diff            @@
##               main    #2241   +/-   ##
=========================================
  Coverage     18.28%   18.28%           
- Complexity     1133     1139    +6     
=========================================
  Files           613      619    +6     
  Lines         17836    17904   +68     
  Branches       2177     2177           
=========================================
+ Hits           3261     3274   +13     
- Misses        14253    14307   +54     
- Partials        322      323    +1

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@ferenc-csaky
ferenc-csaky force-pushed the feat/sparksql-dialect branch 2 times, most recently from 99114b2 to e193fc2 Compare July 31, 2026 11:59
@ferenc-csaky
ferenc-csaky changed the base branch from main to feat/query-engine-validation July 31, 2026 12:02
@ferenc-csaky ferenc-csaky changed the title feat: Add Spark SQL as a query engine feat: Add Spark SQL as a shallow query engine Jul 31, 2026
@ferenc-csaky ferenc-csaky added the enhancement New feature or request label Jul 31, 2026
@ferenc-csaky ferenc-csaky added this to the 0.11.0 milestone Jul 31, 2026
Base automatically changed from feat/query-engine-validation to main July 31, 2026 23:49
@ferenc-csaky
ferenc-csaky force-pushed the feat/sparksql-dialect branch 2 times, most recently from b453951 to 3c7be08 Compare August 3, 2026 14:17
mateczagany
mateczagany reviewed Aug 3, 2026
View reviewed changes
Comment thread ...qrl-testing-integration/src/test/resources/engine-validation/package-sparksql-no-server.json
Comment thread sqrl-planner/src/main/java/com/datasqrl/calcite/dialect/ExtendedSparkSqlDialect.java
@ferenc-csaky
ferenc-csaky force-pushed the feat/sparksql-dialect branch 2 times, most recently from 1175d37 to 09a3f88 Compare August 4, 2026 10:22
@ferenc-csaky
ferenc-csaky force-pushed the feat/sparksql-dialect branch 3 times, most recently from f7fd064 to 9b4d051 Compare August 4, 2026 15:51
@ferenc-csaky
feat: Add Spark SQL as a query engine
67d4d98 
# Conflicts:
#	.gitignore
#	sqrl-planner/src/main/java/com/datasqrl/engine/database/QueryEngine.java
#	sqrl-planner/src/main/java/com/datasqrl/engine/database/relational/DuckDbStatementFactory.java
#	sqrl-planner/src/main/java/com/datasqrl/engine/database/relational/SnowflakeEngine.java
#	sqrl-testing/sqrl-testing-integration/src/test/resources/snapshots/com/datasqrl/EngineValidationTest/package-fail.txt
@ferenc-csaky
ferenc-csaky force-pushed the feat/sparksql-dialect branch from 9b4d051 to 67d4d98 Compare August 5, 2026 05:37
@ferenc-csaky
ferenc-csaky merged commit bf799f9 into main Aug 5, 2026
16 checks passed
@ferenc-csaky
ferenc-csaky deleted the feat/sparksql-dialect branch August 5, 2026 06:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mateczagany mateczagany mateczagany approved these changes

Assignees

No one assigned

Labels

enhancement New feature or request

Projects

None yet

Milestone

0.11.0

Development

Successfully merging this pull request may close these issues.

2 participants

@ferenc-csaky @mateczagany